Modern mobile devices are technological marvels, containing powerful computing, communication, interaction and sensing capabilities within a compact, lightweight package. A large and continuously increasing number of newer sensor inputs are available to mobile devices, including accelerometers, gyroscopes, magnetometers, proximity sensors, GPS sensors, touch screens, light sensors, speakers, multiple microphones, multiple cameras and even barometers. They enable a wealth of varied and diverse means of use, most of which are relatively unexplored. Yet user interaction with these devices still follows rather conventional means despite the advent of touch screen-based user interfaces, using old paradigms such as pointers, icons, keyboards, files and text-based interactions.
Hence there is an opportunity to enable advanced user interactions and improved experiences by using more natural means using the wealth of sensory and contextual information provided by various on-device sensors and the computational, communication and other resources available to modern devices.
On the other hand, the vast majority of the current information and computing infrastructure, mainly current operating systems, operating environments, graphical user interfaces, various native applications, browser-based web applications, the Internet and the World Wide Web, are still limited to older, non touch-based, primarily text-driven user interfaces. Hence there is also a need to bridge the natural user interactions enabled by modern mobile devices with the conventional interactions primarily based on text, icons and menus supported by existing interfaces.